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Survey of State Procedures for the Validation » 
of Educational Programs 



EXECUTIVE SUMMARY 



This survey study was designed to identify: (1) the extent to which 
procedures to validate exemplary programs are being used by state departments 
of education in the f ift)^jtates; (2) the nature of the validation processes 
^in use; (3) state implementation procedures for validation and the scope of 
. state validation efforts to date; (4) state organizational arrangements for 
validation and the nature of state support for the 'dissemination of validated 
programs,- iid (5) the extent of state col.lal^prative validation activities. 
The study also identified state concerns and recommendations pertinent ^o 
validation. 

The study was conducted in 1980-1981 as a collaborative effort of the 
Regional Exchanges (Rxs) and thej Research and Development Interpretation 
Service (RDIS) of the natipnntfide Research and Development Exchange (RDx), 
funded by the Nation^^l Institute !of Education. RDIS staff were primarily 
resporlsible for the design' and overall man^ement of the survey. Each\of the 
Regional Exchanges assisted with the collection of data from state staff in " 
their respective regions and also served as. members of the RDx Task Force on 
Validation. Staff from RDIS, located at CEMREL, Inc., from Appalachia 
Educational Laboratory' (AEL), and from Research for Better Schools (RBS) 
shared in the preparation of the survey report. 

Preliminary data collection generally occurred in the period of 
February-August 1980. Initial -summaries of the data were examined at a ' 
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task force meeting in October 1980. "Follow-up cjata collection and 
verification of data, to assure a common data base aqross Rx regions, occurred 
in .January-February 19ffl. The first'draft of the report^as reviewed for 
completeness^^ clarity, and utility by a nationally representative panel pf 
state staff in Jdly 1981. The report was revised to accommddate the majority 
of the panel's suggestions. ^ 

. The survey results are summarized in Chart 1, an Overview of Currer^t State 
Validation Practices, which is broken into two parts. Part 1 identifies the 
validation process used by each state and state implementation procedures and 
valiation efforts to date. Part 2 identifies state organizational 
arrangements and support for dissemination of validated programs. To 
facilitate a better understanding of .these charts, brief descriptions of the 
Identification, Validation^ Dissemination (IVD) process, the Joint 
Disseminatiort Review Panel (JDRP) process^ and the .Sharing. Business Success 
process follow the chart. 
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Exlent^of State Participation in Validation Activities 
Of the fifty states surveyed, the greal majority (N=45) have some form 
of establ^ished procedure for validating promising educational 
practices. Alabama, Hawaii, Louisiana, Mississippi, and, Nevada 
reported that they do not have a procedure. 

General Processes Employee] by States 

Approximately two-thirds^(N»29) of the states that have validation 
programs (N=45) employ either the IVD procedure (N=23) 
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or some modification of the IVii procedure (N=6). Typical ^ - 

modi fic1?f ions of IVD involve the u^e of in-state site reviewers to 
reduce costs and/or the use of state r^eview panels in conjunction with 
.the site visits. Roughly a third of the\states {N=16) reported that 
they use their own state-developed validation processes. The/ 
state-developed processes differ from IVD With respect to the speqific 
criteria and operational procedures useff to verify the effectiveness 
and transportability of the promising, practices. Two of the states 
with their own processes (New Hampshire and Pennsylvania) viewed them 
as modifications of the JDRP pro^dess. 



State Training^ Implementation Procedures, Schedules^ and Validation 
Efforts \ ^ 

Training for validation team members is a fundamental implemen/tation 
(^oncern. Eighteen states reported that IVD sponsored traininc 
sessions constitute the primary source of training for the1r\]^ 
validation team members. Twenty-five* states conduct their owrt 
training sessions. ,The training varies from one- or two-hour briefing 
sessions-on a staters validation criteria to one- or two-day \^orkshops 
on validation criteria, sinlilations, procedures^ and skil-ls. Two- 
states, Colorado and Maryland, use both IVff gind state-developed 
training. • • ^ • ' . 

In the great majority of cases (N=39) the actual implementation of the 
sta1;e validation process involves some form of prescreening to 



determine if'a project is ready for validation. This prescreening 
activity is an essential, but not widely publicized', aspect of state 
validation practices. Significant variations exist in the. number, 
composition, and title of the state screening committees (teams, ■ 
panels, advisory councils, review' committees, etc.). Their primary 
aim, however, is to avoid the expense and embarrassment that occur 
when- site' visits are made to districts not fully prepared for • . 
validation. Following prescreening, most states (N=43) 'conduct site 
visits to verify, first-hand, projects' status on the state's 
validation Criteria. Ag^in, significant variation occurs^ in the 
length of the site visits (one to thre| days), the number of team 
members, (one to five), and the composition of the teams- (state/ltfcal 
mix arid in/out-of-state mix). Half of the states that conduct site 
visits use only in-state members on their site visit validation teams 
(N«20) primarily for economic reasons. Among the states that do use 
out H)f -state validation team members, thereN^s a growing^ trend to use 
a reduced number of out-of-state stWf, again for economic rea^ions. ^ 
In the majority of the above cases the site-team' report /r€\gpnwendat ion 
to "validate" or ."not validate" a project constitutes -the state's- 
bottom-line procedure for valid^iting exemplary programs— "that is, the 
validation team 2i the decision-making body. 

~ \ ■ 

In three states, howe^^^r--^lifornia, Kentucky, and Pennsylvania— the 
decision-making procedure that is used to validate exemplary 



programs consists of.-a majority approval by a panel pf reviewers. As 
might be expected, there are variations among these states in panel 
numbers and composition, ^in state validation criteria, and in the use 
of site-visit data to elucidate further the panel's decisions. In 
California and Pennsylvania site visits are conducted^ only if deemed 
necessary, to' prpvide the panel with, additional tlata. In Kentucky^ they 
are built .into the state process. State schedules for program 
validation also vary. Approximately 24. states have set schedules 
where applications for validation are reviewed and state validation 
procedures are initiated only at a specified j^ime each year. Twenty 
states report that, they hav6 open schedules and either validate 
projects on a continuous year-round basis or in groups at irregular 
intervals at state staff convenience and/or on demand. 

♦ ' • . 

• It. also^ bears noting that only .a quarter of th6 states (N=12][^ offer 

systematic^ proactive, assistance to projects with evaluation tasks 

related to*^ the. validation requirements. Most states inform projects , 

jthat fail validation (ff their weaknesses vis a'* vis the state 

validation criteria and provide informal evaluation assistance or 

/ ' - 

guidance on request. ' ^ . , , 

Finally, -there appears to be a trend toward requiring validated 
projects to be revalidated ^fter a set period of time. Seven 
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states currently have revalidation procedures in operation 'and another 
ten stattes are considering the initiation of program revalidation \ 
procedures^ ; 



.The above summary provides an overview of ex^tant state implementation 
procedures for validating exemplary T)rograms. The actual liunfcer of 
programs reviewed and validated is described next. * 

Roughly thre^-fourths of the states (N=32) involvid in validating' ^ 
exert^)lary programs review between 1 and 10 projects per year. Only • 
twelve states review 11 or more projects per year. Of these 
approximately three- fourths of the* stages (N=29) -validate between 1 
and 5 programs per year. Another nine states validate .from 6-10 
projects per year and six states validate. 11 or more per year. Based 
on conservative mid-range estimates, therefore, betwe.en,200 and. 250 
state exemplary projects are validated annually. in the United States. 



State Organizational Arrangements and Support for the Dissemination' of 
Validated Programs 

'Further examination of the Overview Chart reveals that in the great 

majority of the. states (N^Sy), state ,IV-C^staff have primary 

responsibility for the implementation of the state's validation 

program.. In seven states the responsibility for implementing the 

» / 
validation process Vests wi,th dissemination unit staff, program 

development staff, or research and* development staff. In the majority 

of the states, one to threie staff, ane required to* manage the state's • 



val ideation effort. In most cases these staff have other • 
responsibilities and work on validation re la|ed .tasks only on a 



y part-time basis. A handiiul, of states, however, involve. up to as many 
as twenty-five ot their staff fn the; validation ^ffort, stnict-ly(on an 
Is-^eeded basis for brief periods of time. 



^ • / , 
overall, there appears to be a fairly consistent. po.l icy among the 

■/ ' ■ • • • ' ■ ' • • 

9 ^ 

States to support financially the 'dissemination and adoptfdrtof 
' exemplary projects once they are validated. Twenty-seven states, over' 
half Of the states .involved iri validation,- reported that they provided 
. support to both the developer/demonstrators ~(D/Ds)''and the 
adopter /adapters (A/As), of validated programs.^ Title IV-C monies, 
/constitute the primary source of support for 0/Ds and A/As. D/Ds are ., 
usually provided the greater amount of supjJont'to assist with the^ - 
reproduction of materials, the conduct of -dissemination activities; 
and the partial support of staff.salaries for diss^ination 
activities. A/A grants most^commonly range betw^n $5,000 and $10,000 
and are intended tp support staff development and other- startnip' 
costs,; such as related materials costs. • ^ - 

The Overview Chart also reve'als that most ^tates, require that D/Ds 

* * » < * * 

'operate at least one year as a demonstratfon site (N=40) and pmide 
materials and general assistance to authorized A/As (N=43)*J Only a. 

.f4w states (N«9); however, require thtt D/Ds actively monitor or" 
evaluate, in some way^ implementation 'of exemplary programs by A/As. 
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^The.^ Overview Chart iUiistrates that most states utilize state 
catalogs, workshops, conferences, and various kinds of print media. t6 
disseminate information about validated programs. 

r 

State Collaborative Activities . ^ 

Close to three-fourths of the suites' reportedly are involved in some 
form of cooperative activities with other''nearl?y states. The 
cooperative activities, however,' center primarily on implementation of 
the valfdation procedure (e.g.-, use of other states' validators^ and 
information sharing activities (exchange iD-f state catalogs, joint 
participation in conferences). Few, if any, states cooperate by the 
actual sharing of^ validated programs. There has been, d^iscuss ion of 
the.-cross-^s^ate use of validated programs. At present political 
barriers and lack of incentives have null^ified prospective activity in 
that direction, . ^ - 



The above. description completes the general overview of state validation 
practices current as of June 1981, as. obtained 'from the RDx survey study. In 
.lafge part the great majority of states reported that they were "bullish" on 
the process of validating exemp^lary programs. The states alsq'shared a number 
♦of concerns and recommendations. ^ \ c> 

The. concerns and recommendations voiced regarding both IVD anc^ JDRP are 
indicative of the states' strong interest in the -validation process 'and should 
be viejj^fed accordingly. . - 
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Concerns * • Us ' 

The, primary concerns centered on three key Issues: (IV the tension 

V 

extant between the extreme rigor of the JDRP process and the 

•variability implicit in states' implementation of the "softer" IVD 

process, resulting in- inconsistent federal and state validation 

policies and concomitant duplication of LEA valixlation effprts; (2) 

• • • . >^ 

the lack of sufficient funding or limitations on funding which inhij)it 

additional development work by D/Ds; adequate monitoring and follow-up 

of D/D dissemination and A/A implementation activities, and more 

widespread' intrastate dissemination' of validated projects; and (3) the 

general observation that the dissemination/diffusion of validated IV-C 

programs is ah apparent add-on in the minds of federal planners,. 

Recommendations 

A number of recommendations or suggestions were made in four -general 
areas. \ 
Coordination and communication were the f oc^i of several 
recommendations: . ^ ^ * 

- There should be more and better communication with federal 
officials in the regional offices. 

Funding should be made .available to encourage contiguous states to 
work together for validating programs. 

- The encouragement of use of one set of cV'iteria by all states 
would permit sufficient reliability to allow easier adoption 
across state lines. 
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IVD criteria should be accepted by all ESEA categorical programs, 
r State/regional /national catalogs of programs should include data 
about use of projects by adoptjars. 

Several ^suggestions concerned JDRP and the ^ scope of JDRP programs : 

- JDRP should look at programs in career education, nutrition 
education, child development, ^ etc. ^ 

- JDRP should j'pclude programs developed\fn non-public schools. 

- JDRP should mandate site vi'sitsl' - !' 

< 

- JDRP should encourage programs to subiriit other than just those 
.that are student ^achievement-oriented. 

- JDRP should examine innovations of ' a program developed by ai^opters. 

- JDRP and IVD should be more closely married. 

Several recommendations concerned funding ; 

- 0/Ds should receive more support for ongoing development < 
activities. 

Funds should be dedicated to encouraging regional activities.- 

- The five-year funding limit should be re-examined. 

' - IV-C funding should be increased as more LEAs becoine involved. 

A number of recommendations centered on diffusion efforts : 

- Catalog^ should be updated and non-functioning programs 
eliminated. NDN files should also be updated. * 

- Adopters should be followed up in a systematic way. 

- Re-training should be provided for adopters periodically. 
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As evidenced\^by the above summary, state interest in and concerns about 
extant procedures for validating exemplary. programs is quite vital. The 
present descriptive study was just a first step, serving to identify and focus 
that interest. '^A logical next step would consist of federal and state 
follow-up regarding the concerns and recommendations cited in this report. In 
addition, it is recommended that funding be strongly considered for 
descriptive and impact studies of state dissemination efforts. The present 
study was limited to describing extant state validation procedures, issues,/ 
ar^d concerns. There is -a pressing need, especially in this period of 
declining resources, for further information about short- arjd ^ong-term D/D 
and A/A behaviors, to identify ways to improve the dissemination and use of - 
validated exemplary programs. 



Chart 1; OVERVIEW OF CURRENT STATE VALIDATION PRACTICES 

Part 1 
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Chart 1. OVERVIEW OF CURRENT STATE, VALIDATION PRACTICES 
u \ Part 1 (continued) 
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Chart 1." OVERVIEW OF CURRENT STATE VALIDATION PRACTICES 
^ Part 2 
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V " Variable 
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SF - State Facilitator 
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The RDx Collaborative Effort'on the Validation of Educational Programs and 
Practices includes four products: ' 

Survey of S tate Procedures for the Validation of Educational Programs , by 
Linda Reed. Ed Patrick, and David Holdzkom. St. Louis, Missouri: C^MREL. Inc.. 
for the R&D Exchange, 1981. . ' » •» 

Survey of S tate Procedures for th6 Validation of Educational Programs. 
Executive Summary, bv Linda Reed. Fri P^frir\,. ^nri n.u<H u>^^^.^.■.^ Louis. 
Missouri: CEMREL, Im:., for the RiO Exchange, 1981. 

The Search for Qualit\ control in Dissemination of Educational Prod ucts and 

Practices: A Look at We Literature and Major Issues , bv Li nda'^^'ReSi: 

St. Louis, Missouri :^EMRhL. InC, R&£) Interpretation Service,- 1981. 

Validation of Education al Programs. Practices and Products; An Annotated 
eibliographv. prepared by Karen Temmen, Mary Ann Isaacs, and Sandra Ruder. 
S^t. Louis, Missouri: CEMREL, Inc., for the R&D Exchange, 1981. 



32 



Research and Development Exchange (ROx) 



Repionai Exchanges (Rx) 



A. 



Appalachia Educational Laboratory (AEL) 
P.O. Box 1348 

Charleston, West Virginia ^25325 
(800) 624-9120 ^ 

Director: Sandra Orletsky 



SWRL Research and Development 
'4665 Lampson Avenue 

Los Alamitos, California 90720 
- (213) 598-7661 

Dtrector: Roger Scott ^ 



CEMREL, Inc. ■ 

3120 59th Street 

St. Louis, Missouri 63139 

(314) 781-2900 

Director*\^Carol Thomas 
McREL 

4709 Belleview 

Kansas City,. Missouri 64112 

(816) 756-2401* 

Director: .Susan Everson 

Northeast Regional Exchange (NEREX) 
Merrimack Education Center 
101 Mill Road 

Chelmsford, Massachusetts 01824 
(617) 256-3985 

Directpr: J, Lynn- Griesemer 

Northwest Regional Educational 

Laboratory (NWREL) 
300 S.w: Sixth Avenue - 
Portland, Oregon 97264 
(503) 248-6800 

Director: Joe^ascarelli 

Research for Better Schools, Inc. (RBS) 
444 North Third Street 
Philadelphia, Pennsylvania' 19123 
(215)^^«l-9300 

Director: Richard McCann 



'Southwest Educational Development 

Laboratory (SEDL) 
211 East Seventh Street 
Austin, Texais 78701 
(512) 476-6861 . 

Director: Preston Kronkosky 
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Central Supporf SeVvices 

System Support 'Service 

Far West Laboratory for Educational 

Research and Deve^opqjent 
1855 Folsom Street 
San Francisco, California 94103 
(415) 565-3179 

Director: Stanley Chow 

RiD Interpretation Service 
CEMREL, Inc. 
3T20 59th Street 

St. Louis, Missouri 63139 - 
(314) 781-2900 

Director: Linda Reed 

Resource and Referral Service 
National Center for Research in . 

Vocational Education 
I960- Kenny Road' 
Columbus, Ohio 43210 
(614) 486-^655 . I • • 

Director: Jay Srfiink 

Dissemination S'ypport Service- 
Northwest Regional Educatioha"l 

Laboratory^ 
300-,S.W. Sixth Avenue ' . 
Portland, Oregon 97204 
(503) 248-6800 • 

Director: Joe Pascarelli ^ . 



